Railway•17mo ago

DNS Error when using Fetch API only in Railway

I wrote a crawler using the crawlbase api. It runs fine in a container on my local machine, but in railway, it seems like im getting some kind of dns error. Container Base: node:18.16-alpine Its just a basic node + typescript application. Here is an example of how im fetching.

    const response = await fetch(
        'https://api.crawlbase.com'
        + qs.stringify({
            url,
            format: 'json',
            token: token,
        }, { encode: true, addQueryPrefix: true })
    );

    const response = await fetch(
        'https://api.crawlbase.com'
        + qs.stringify({
            url,
            format: 'json',
            token: token,
        }, { encode: true, addQueryPrefix: true })
    );

The error looks something like this

TypeError: fetch failed
at Object.fetch (node:internal/deps/undici/undici:11457:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
cause: Error: getaddrinfo EAI_AGAIN api.crawlbase.com
at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
errno: -3001,
code: 'EAI_AGAIN',
syscall: 'getaddrinfo',
hostname: 'api.crawlbase.com'
}
}

TypeError: fetch failed
at Object.fetch (node:internal/deps/undici/undici:11457:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
cause: Error: getaddrinfo EAI_AGAIN api.crawlbase.com
at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
errno: -3001,
code: 'EAI_AGAIN',
syscall: 'getaddrinfo',
hostname: 'api.crawlbase.com'
}
}

Not really sure why this is and wanted to make sure there aren't any internal dns gotchas

Solution:

Jump to solution

10 Replies

Percy•17mo ago

Project ID: 764096d4-61ee-4d50-af38-6a9a0eda6dd7

EricOP•17mo ago

764096d4-61ee-4d50-af38-6a9a0eda6dd7

Brody•17mo ago

dns lookup for that domain is taking over 5 seconds on railway for some reason https://utilities.up.railway.app/dns-lookup?value=api.crawlbase.com&type=ip

EricOP•17mo ago

Oh.... that is super weird. It would be nice it was that slow, but maybe I can do something in code to look up that dns record with a longer timeout and then cache it. Never missed with that in node so will have to look into it I guess

Brody•17mo ago

you can create a new resolver with a custom timeout set, but can you pass that into the fetch request? or just create a resolver that uses 1.1.1.1 https://utilities.up.railway.app/dns-lookup?value=api.crawlbase.com&type=ip&dns=1.1.1.1

EricOP•17mo ago

Oh, that is awesome. Thank you. I was just about to test my solution. Didn't know that 1.1.1.1 was faster. I had to switch from fetch to to the https node library so that I could pass in a lookup function. Then I juse used a resolver with a large timeout there. I'm also going to make sure that the resolver uses 1.1.1.1 as the dns server so its faster. Going to post code and mark as solved if this works

Brody•17mo ago

no timeout needed if you use 1.1.1.1 🙂

EricOP•17mo ago

It seems like this works. You might be able to get away with just using dns.setServers(['1.1.1.1']) and the using fetch. I couldn't find a whole ton of documentation on that. I ended up just ended up writing this request.ts file. Timeout is probably optional as well, but I wanted to make sure so I left it in.

import https from 'https';
import dns from 'dns';

const resolver = new dns.Resolver({
    timeout: 10000
});
resolver.setServers([
    "1.1.1.1", // 1.1.1.1 resolves much faster than the default when running in railway
    "1.0.0.1",
    "8.8.8.8",
    "8.8.4.4"
])

const lookup = (
    hostname: string,
    _options: dns.LookupOneOptions,
    callback: (err: NodeJS.ErrnoException | null, address: string, family: number) => void
) => resolver.resolve(hostname, 'A', (err, addresses) => callback(err, (addresses.length) ? addresses[0] : "", 4))

export const request = async (options: https.RequestOptions) => {
    return new Promise<{ body: string, status: number | undefined }>((resolve, reject) => {
        const req = https.request({ ...options, lookup }, (res) => {
            let body = '';
            res.on('data', (chunk) => {
                body += chunk;
            });

            res.on('end', () => {
                const status = res.statusCode;
                res.destroy();

                resolve({
                    body,
                    status: status
                });
            });

            res.on('error', (err) => {
                reject(err);
            });
        });

        req.on('error', (err) => {
            reject(err);
        });

        req.end();
    })
}

import https from 'https';
import dns from 'dns';

const resolver = new dns.Resolver({
    timeout: 10000
});
resolver.setServers([
    "1.1.1.1", // 1.1.1.1 resolves much faster than the default when running in railway
    "1.0.0.1",
    "8.8.8.8",
    "8.8.4.4"
])

const lookup = (
    hostname: string,
    _options: dns.LookupOneOptions,
    callback: (err: NodeJS.ErrnoException | null, address: string, family: number) => void
) => resolver.resolve(hostname, 'A', (err, addresses) => callback(err, (addresses.length) ? addresses[0] : "", 4))

export const request = async (options: https.RequestOptions) => {
    return new Promise<{ body: string, status: number | undefined }>((resolve, reject) => {
        const req = https.request({ ...options, lookup }, (res) => {
            let body = '';
            res.on('data', (chunk) => {
                body += chunk;
            });

            res.on('end', () => {
                const status = res.statusCode;
                res.destroy();

                resolve({
                    body,
                    status: status
                });
            });

            res.on('error', (err) => {
                reject(err);
            });
        });

        req.on('error', (err) => {
            reject(err);
        });

        req.end();
    })
}

Then I call it like this

    const response = await request({
        hostname: 'api.crawlbase.com',
        path: '/' + qs.stringify({
            url,
            format: 'json',
            token: token,
        }, { encode: true, addQueryPrefix: true }),
        method: 'GET',
    });
    const json = JSON.parse(response.body);

    const response = await request({
        hostname: 'api.crawlbase.com',
        path: '/' + qs.stringify({
            url,
            format: 'json',
            token: token,
        }, { encode: true, addQueryPrefix: true }),
        method: 'GET',
    });
    const json = JSON.parse(response.body);

Thanks for your help @Brody dns timeouts are fixed and its way faster after setting the servers

EricOP•17mo ago

oops. I messed up linking the solution correctly. I see how that works now but don't know how to fix it

Brody•17mo ago

don't worry about it I'm glad I could help

Gaming

Programming

DNS Error when using Fetch API only in Railway