Eric Comments - Answer Overflow

It seems like this works. You might be able to get away with just using dns.setServers(['1.1.1.1']) and the using fetch. I couldn't find a whole ton of documentation on that. I ended up just ended up writing this request.ts file. Timeout is probably optional as well, but I wanted to make sure so I left it in.

import https from 'https';
import dns from 'dns';

const resolver = new dns.Resolver({
    timeout: 10000
});
resolver.setServers([
    "1.1.1.1", // 1.1.1.1 resolves much faster than the default when running in railway
    "1.0.0.1",
    "8.8.8.8",
    "8.8.4.4"
])

const lookup = (
    hostname: string,
    _options: dns.LookupOneOptions,
    callback: (err: NodeJS.ErrnoException | null, address: string, family: number) => void
) => resolver.resolve(hostname, 'A', (err, addresses) => callback(err, (addresses.length) ? addresses[0] : "", 4))

export const request = async (options: https.RequestOptions) => {
    return new Promise<{ body: string, status: number | undefined }>((resolve, reject) => {
        const req = https.request({ ...options, lookup }, (res) => {
            let body = '';
            res.on('data', (chunk) => {
                body += chunk;
            });

            res.on('end', () => {
                const status = res.statusCode;
                res.destroy();

                resolve({
                    body,
                    status: status
                });
            });

            res.on('error', (err) => {
                reject(err);
            });
        });

        req.on('error', (err) => {
            reject(err);
        });

        req.end();
    })
}

import https from 'https';
import dns from 'dns';

const resolver = new dns.Resolver({
    timeout: 10000
});
resolver.setServers([
    "1.1.1.1", // 1.1.1.1 resolves much faster than the default when running in railway
    "1.0.0.1",
    "8.8.8.8",
    "8.8.4.4"
])

const lookup = (
    hostname: string,
    _options: dns.LookupOneOptions,
    callback: (err: NodeJS.ErrnoException | null, address: string, family: number) => void
) => resolver.resolve(hostname, 'A', (err, addresses) => callback(err, (addresses.length) ? addresses[0] : "", 4))

export const request = async (options: https.RequestOptions) => {
    return new Promise<{ body: string, status: number | undefined }>((resolve, reject) => {
        const req = https.request({ ...options, lookup }, (res) => {
            let body = '';
            res.on('data', (chunk) => {
                body += chunk;
            });

            res.on('end', () => {
                const status = res.statusCode;
                res.destroy();

                resolve({
                    body,
                    status: status
                });
            });

            res.on('error', (err) => {
                reject(err);
            });
        });

        req.on('error', (err) => {
            reject(err);
        });

        req.end();
    })
}

Then I call it like this

    const response = await request({
        hostname: 'api.crawlbase.com',
        path: '/' + qs.stringify({
            url,
            format: 'json',
            token: token,
        }, { encode: true, addQueryPrefix: true }),
        method: 'GET',
    });
    const json = JSON.parse(response.body);

    const response = await request({
        hostname: 'api.crawlbase.com',
        path: '/' + qs.stringify({
            url,
            format: 'json',
            token: token,
        }, { encode: true, addQueryPrefix: true }),
        method: 'GET',
    });
    const json = JSON.parse(response.body);

Thanks for your help @Brody dns timeouts are fixed and its way faster after setting the servers

15 replies

RRailway

•Created by Eric on 7/12/2023 in #✋｜help

DNS Error when using Fetch API only in Railway

Oh, that is awesome. Thank you. I was just about to test my solution. Didn't know that 1.1.1.1 was faster. I had to switch from fetch to to the https node library so that I could pass in a lookup function. Then I juse used a resolver with a large timeout there. I'm also going to make sure that the resolver uses 1.1.1.1 as the dns server so its faster. Going to post code and mark as solved if this works

15 replies

RRailway

•Created by Eric on 7/12/2023 in #✋｜help

DNS Error when using Fetch API only in Railway

Oh.... that is super weird. It would be nice it was that slow, but maybe I can do something in code to look up that dns record with a longer timeout and then cache it. Never missed with that in node so will have to look into it I guess

15 replies

RRailway

•Created by Eric on 7/12/2023 in #✋｜help

DNS Error when using Fetch API only in Railway

764096d4-61ee-4d50-af38-6a9a0eda6dd7

15 replies

Gaming

Programming