Railway•2y ago

Private networking - webhook error

I'm working on a new callback pattern for realtime GraphQL subscriptions that utilizes a webhook like pattern. I want to use the internal networking in railway to secure that callback url, but I'm getting an error:

request to http://{private}.railway.internal:6607/callback/71d75f513901349ee7dfe50054409d4b019887cdd9c6f79e41b8ee752d293ae7 failed, reason: getaddrinfo ENOTFOUND hosted-router.railway.internal

request to http://{private}.railway.internal:6607/callback/71d75f513901349ee7dfe50054409d4b019887cdd9c6f79e41b8ee752d293ae7 failed, reason: getaddrinfo ENOTFOUND hosted-router.railway.internal

I'm currently listing my public url as: http://{private}.railway.internal:${env.PORT:-4000}/callback Is there anything jumping out why this wouldn't be working? I already have everything working successfully when I use the public domains avaialble.

41 Replies

Percy•2y ago

Project ID: 38b0d466-9168-4f9f-bf7f-c09650f565e8

__watsonOP•2y ago

38b0d466-9168-4f9f-bf7f-c09650f565e8

Brody•2y ago

the private networking interface takes about 100ms to become bound, and another 2-3 seconds for private dns resolution to come online, any internal network requets during this time will very likely fail

__watsonOP•2y ago

Does it mean we need ~3s on pod startup or for each of the network calls?

Brody•2y ago

if they are internal network calls, yes, a 3 second grace period is needed, or auto retries is an option too railway will be working to shorten or eliminate the need for this pause

__watsonOP•2y ago

Okay I've tried adding some stuff here but I'm still getting errors. I added some code to wait for the service to be ready:

await new Promise((resolve) => {
  const interval = setInterval(async () => {
    try {
      const result = await fetch(
        `http://{private}.railway.internal:7718`
      );
      console.log(`Result: ${JSON.stringify(result)}`);
      if (result) {
        console.log(`Pass`);
        resolve("foo");
        clearInterval(interval);
      }
    } catch (err) {
      console.log(JSON.stringify(err));
    }
  }, 1000);
});

await new Promise((resolve) => {
  const interval = setInterval(async () => {
    try {
      const result = await fetch(
        `http://{private}.railway.internal:7718`
      );
      console.log(`Result: ${JSON.stringify(result)}`);
      if (result) {
        console.log(`Pass`);
        resolve("foo");
        clearInterval(interval);
      }
    } catch (err) {
      console.log(JSON.stringify(err));
    }
  }, 1000);
});

I hardcoded the port that my other service is in and I'm seeing this error in my other service (project id: 19b30993-43c6-4719-b45a-81d55968ec20):

{"cause":{"errno":-3008,"code":"ENOTFOUND","syscall":"getaddrinfo","hostname":"hosted-router.railway.internal"}}

{"cause":{"errno":-3008,"code":"ENOTFOUND","syscall":"getaddrinfo","hostname":"hosted-router.railway.internal"}}

I figured I should get any positive http response (even 404 is fine) and then it's good to continue my request chain

Brody•2y ago

so uh whats with {private} is that just you hiding information?

__watsonOP•2y ago

I guess? I mean yes, is it hiding anythhing really?

Brody•2y ago

wdym you guess? lol is that what your code actually has written in it?

__watsonOP•2y ago

I have this http://hosted-router.railway.internal:7718

Brody•2y ago

okay so what was the point of {private}

__watsonOP•2y ago

So I didn't put hosted-router in discord, but I'll just change that later

Brody•2y ago

are you using nixpacks

__watsonOP•2y ago

Currently only the defaults when I deploy, I see Nixpacks in my settings actually the service that I'm exposing the internal domain on is docker the service it's being reached from is using Nixpacks

Brody•2y ago

everything is docker but the image is built with nixpacks do you have internal networking enabled

__watsonOP•2y ago

__watsonOP•2y ago

that right?

Brody•2y ago

i assume you are just seeing that json error log indefinably?

__watsonOP•2y ago

yup, still going in the logs

Brody•2y ago

and that code is from a nodejs server

__watsonOP•2y ago

yup

Brody•2y ago

well im stumped

__watsonOP•2y ago

This stuff is hard

Brody•2y ago

its always just worked for me lol

__watsonOP•2y ago

I know that feeling so well 🤣

Brody•2y ago

wanna try awaiting a setTimeout so that it acts as a blocking pause before any network requests are done?

__watsonOP•2y ago

yeah like 3s?

Brody•2y ago

yes, max time ive seen is 3s

Brody•2y ago

average time is 1.5s but 3s is a good safe time

__watsonOP•2y ago

no luck

Brody•2y ago

i dont see a pause in the timestamps? also, node 20??

__watsonOP•2y ago

🤷‍♂️ I just say above 18

  "engines": {
    "node": ">=18.0"
  },

  "engines": {
    "node": ">=18.0"
  },

Sorry I misunderstood and put the sleep before the timestamps, not in them. I'll change that

Brody•2y ago

where did you put the pause? sleep 3 && npm run start

__watsonOP•2y ago

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
await sleep(3000);

await new Promise((resolve) => {
  const interval = setInterval(async () => {
    try {
      const result = await fetch(
        `http://hosted-router.railway.internal:7718`
      );
      console.log(`Result: ${JSON.stringify(result)}`);
      if (result) {
        console.log(`Pass`);
        resolve("foo");
        clearInterval(interval);
      }
    } catch (err) {
      console.log(JSON.stringify(err));
    } finally {
      await sleep(3000);
    }
  }, 1000);
});

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
await sleep(3000);

await new Promise((resolve) => {
  const interval = setInterval(async () => {
    try {
      const result = await fetch(
        `http://hosted-router.railway.internal:7718`
      );
      console.log(`Result: ${JSON.stringify(result)}`);
      if (result) {
        console.log(`Pass`);
        resolve("foo");
        clearInterval(interval);
      }
    } catch (err) {
      console.log(JSON.stringify(err));
    } finally {
      await sleep(3000);
    }
  }, 1000);
});

👆 that's just in my code, it's a web server that is already started up before the request issue

Brody•2y ago

and same issue? as a last ditch attempt, add ENABLE_ALPINE_PRIVATE_NETWORKING set to true in your service variables

__watsonOP•2y ago

Okay there was some success in that, I get a response error:

{
    "data": null,
    "errors": [
        {
            "message": "HTTP fetch failed from 'playback': HTTP fetch failed from 'playback': 503: Service Unavailable",
            "path": [],
            "extensions": {
                "code": "SUBREQUEST_HTTP_ERROR",
                "service": "playback",
                "reason": "HTTP fetch failed from 'playback': 503: Service Unavailable"
            }
        }
    ]
}

{
    "data": null,
    "errors": [
        {
            "message": "HTTP fetch failed from 'playback': HTTP fetch failed from 'playback': 503: Service Unavailable",
            "path": [],
            "extensions": {
                "code": "SUBREQUEST_HTTP_ERROR",
                "service": "playback",
                "reason": "HTTP fetch failed from 'playback': 503: Service Unavailable"
            }
        }
    ]
}

This is now something else I need to look into that might not be railway, since this is the Apollo Router receiving a response from a downstream service and that is the actual gql response coming back

Brody•2y ago

one more thing in the hosted-router.railway.internal service variables do you have a fixed PORT set?

__watsonOP•2y ago

I do not, I just have hardcoded in the subgraph I'm testing that port hardcoded everything else listens on the $PORT environment variable (which I don't set) (but I might need to)

Brody•2y ago

since railway assigns it randomly, that means it makes it hard to do an internal request to a service on a random PORT so yeah might want to set that to a fixed value in the service variables

__watsonOP•2y ago

ya good call, I will do that for sure I at least have something more to investigate and I'll report back my findings end of day. I'm hoping to trim that code down into a small block I can reference and ideally delete in the future when y'all get to that point ❤️

Brody•2y ago

i really do hope railway is able to make private network initiation instant so theres no need for these hacky work arounds. dont tell them i said this, but i dont think they sould have brought privnets out to public and enabled them by default with this flaw

Gaming

Programming

Private networking - webhook error