Railway•13mo ago

Private networking - webhook error

I'm working on a new callback pattern for realtime GraphQL subscriptions that utilizes a webhook like pattern. I want to use the internal networking in railway to secure that callback url, but I'm getting an error:

request to http://{private}.railway.internal:6607/callback/71d75f513901349ee7dfe50054409d4b019887cdd9c6f79e41b8ee752d293ae7 failed, reason: getaddrinfo ENOTFOUND hosted-router.railway.internal

request to http://{private}.railway.internal:6607/callback/71d75f513901349ee7dfe50054409d4b019887cdd9c6f79e41b8ee752d293ae7 failed, reason: getaddrinfo ENOTFOUND hosted-router.railway.internal

I'm currently listing my public url as: http://{private}.railway.internal:${env.PORT:-4000}/callback Is there anything jumping out why this wouldn't be working? I already have everything working successfully when I use the public domains avaialble.

41 Replies

Percy•13mo ago

Project ID: 38b0d466-9168-4f9f-bf7f-c09650f565e8

__watson•13mo ago

38b0d466-9168-4f9f-bf7f-c09650f565e8

Brody•13mo ago

the private networking interface takes about 100ms to become bound, and another 2-3 seconds for private dns resolution to come online, any internal network requets during this time will very likely fail

__watson•13mo ago

Does it mean we need ~3s on pod startup or for each of the network calls?

Brody•13mo ago

if they are internal network calls, yes, a 3 second grace period is needed, or auto retries is an option too railway will be working to shorten or eliminate the need for this pause

__watson•13mo ago

Okay I've tried adding some stuff here but I'm still getting errors. I added some code to wait for the service to be ready:

await new Promise((resolve) => {
  const interval = setInterval(async () => {
    try {
      const result = await fetch(
        `http://{private}.railway.internal:7718`
      );
      console.log(`Result: ${JSON.stringify(result)}`);
      if (result) {
        console.log(`Pass`);
        resolve("foo");
        clearInterval(interval);
      }
    } catch (err) {
      console.log(JSON.stringify(err));
    }
  }, 1000);
});

await new Promise((resolve) => {
  const interval = setInterval(async () => {
    try {
      const result = await fetch(
        `http://{private}.railway.internal:7718`
      );
      console.log(`Result: ${JSON.stringify(result)}`);
      if (result) {
        console.log(`Pass`);
        resolve("foo");
        clearInterval(interval);
      }
    } catch (err) {
      console.log(JSON.stringify(err));
    }
  }, 1000);
});

I hardcoded the port that my other service is in and I'm seeing this error in my other service (project id: 19b30993-43c6-4719-b45a-81d55968ec20):

{"cause":{"errno":-3008,"code":"ENOTFOUND","syscall":"getaddrinfo","hostname":"hosted-router.railway.internal"}}

{"cause":{"errno":-3008,"code":"ENOTFOUND","syscall":"getaddrinfo","hostname":"hosted-router.railway.internal"}}

I figured I should get any positive http response (even 404 is fine) and then it's good to continue my request chain

Brody•13mo ago

so uh whats with {private} is that just you hiding information?

__watson•13mo ago

I guess? I mean yes, is it hiding anythhing really?

Brody•13mo ago

wdym you guess? lol is that what your code actually has written in it?

__watson•13mo ago

I have this http://hosted-router.railway.internal:7718

Brody•13mo ago

okay so what was the point of {private}

__watson•13mo ago

So I didn't put hosted-router in discord, but I'll just change that later

Brody•13mo ago

are you using nixpacks

__watson•13mo ago

Currently only the defaults when I deploy, I see Nixpacks in my settings actually the service that I'm exposing the internal domain on is docker the service it's being reached from is using Nixpacks

Brody•13mo ago

everything is docker but the image is built with nixpacks do you have internal networking enabled

__watson•13mo ago

__watson•13mo ago

that right?

Brody•13mo ago

i assume you are just seeing that json error log indefinably?

__watson•13mo ago

yup, still going in the logs

Brody•13mo ago

and that code is from a nodejs server

__watson•13mo ago

yup

Brody•13mo ago

well im stumped

__watson•13mo ago

This stuff is hard

Brody•13mo ago

its always just worked for me lol

__watson•13mo ago

I know that feeling so well 🤣

Brody•13mo ago

wanna try awaiting a setTimeout so that it acts as a blocking pause before any network requests are done?

__watson•13mo ago

yeah like 3s?

Brody•13mo ago

yes, max time ive seen is 3s

Brody•13mo ago

average time is 1.5s but 3s is a good safe time

__watson•13mo ago

no luck

Brody•13mo ago

i dont see a pause in the timestamps? also, node 20??

__watson•13mo ago

🤷‍♂️ I just say above 18

  "engines": {
    "node": ">=18.0"
  },

  "engines": {
    "node": ">=18.0"
  },

Sorry I misunderstood and put the sleep before the timestamps, not in them. I'll change that

Brody•13mo ago

where did you put the pause? sleep 3 && npm run start

__watson•13mo ago

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
await sleep(3000);

await new Promise((resolve) => {
  const interval = setInterval(async () => {
    try {
      const result = await fetch(
        `http://hosted-router.railway.internal:7718`
      );
      console.log(`Result: ${JSON.stringify(result)}`);
      if (result) {
        console.log(`Pass`);
        resolve("foo");
        clearInterval(interval);
      }
    } catch (err) {
      console.log(JSON.stringify(err));
    } finally {
      await sleep(3000);
    }
  }, 1000);
});

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
await sleep(3000);

await new Promise((resolve) => {
  const interval = setInterval(async () => {
    try {
      const result = await fetch(
        `http://hosted-router.railway.internal:7718`
      );
      console.log(`Result: ${JSON.stringify(result)}`);
      if (result) {
        console.log(`Pass`);
        resolve("foo");
        clearInterval(interval);
      }
    } catch (err) {
      console.log(JSON.stringify(err));
    } finally {
      await sleep(3000);
    }
  }, 1000);
});

👆 that's just in my code, it's a web server that is already started up before the request issue

Brody•13mo ago

and same issue? as a last ditch attempt, add ENABLE_ALPINE_PRIVATE_NETWORKING set to true in your service variables

__watson•13mo ago

Okay there was some success in that, I get a response error:

{
    "data": null,
    "errors": [
        {
            "message": "HTTP fetch failed from 'playback': HTTP fetch failed from 'playback': 503: Service Unavailable",
            "path": [],
            "extensions": {
                "code": "SUBREQUEST_HTTP_ERROR",
                "service": "playback",
                "reason": "HTTP fetch failed from 'playback': 503: Service Unavailable"
            }
        }
    ]
}

{
    "data": null,
    "errors": [
        {
            "message": "HTTP fetch failed from 'playback': HTTP fetch failed from 'playback': 503: Service Unavailable",
            "path": [],
            "extensions": {
                "code": "SUBREQUEST_HTTP_ERROR",
                "service": "playback",
                "reason": "HTTP fetch failed from 'playback': 503: Service Unavailable"
            }
        }
    ]
}

This is now something else I need to look into that might not be railway, since this is the Apollo Router receiving a response from a downstream service and that is the actual gql response coming back

Brody•13mo ago

one more thing in the hosted-router.railway.internal service variables do you have a fixed PORT set?

__watson•13mo ago

I do not, I just have hardcoded in the subgraph I'm testing that port hardcoded everything else listens on the $PORT environment variable (which I don't set) (but I might need to)

Brody•13mo ago

since railway assigns it randomly, that means it makes it hard to do an internal request to a service on a random PORT so yeah might want to set that to a fixed value in the service variables

__watson•13mo ago

ya good call, I will do that for sure I at least have something more to investigate and I'll report back my findings end of day. I'm hoping to trim that code down into a small block I can reference and ideally delete in the future when y'all get to that point ❤️

Brody•13mo ago

i really do hope railway is able to make private network initiation instant so theres no need for these hacky work arounds. dont tell them i said this, but i dont think they sould have brought privnets out to public and enabled them by default with this flaw

Gaming

Programming

Private networking - webhook error